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Abstract. This is a second paper in a series devoted to the minimal unitary representation 
of 0(p,q). By explicit methods from conformal geometry of pseudo Riemannian manifolds, 
we find the branching law corresponding to restricting the minimal unitary representation 
to natural symmetric subgroups. In the case of purely discrete spectrum we obtain the 
full spectrum and give an explicit Parseval-Plancherel formula, and in the general case we 
construct an infinite discrete spectrum. 
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Introduction 

This is the second in a series of papers devoted to the analysis of the minimal repre- 
sentation w p,q of 0(p : q). We refer to [23] for a general introduction; also the numbering 
of the sections is continued from that paper, and we shall refer back to sections there. 
However, the present paper may be read independently from [23] , and and its main object 
is to study the branching law for the minimal unitary representation w p,q from analytic 
and geometric point of view. Namely, we shall find by explicit means, coming from 
conformal geometry, the restriction of w p,q with respect to the symmetric pair 



(G,G') = (0( M ),0(p' l9 ')xO(p", g ")). 

If one of the factors in G' is compact, then the spectrum is discrete (see Theorem 4.2 also 
for an opposite implication), and we find the explicit branching law; when both factors 
are non-compact, there will still (generically) be an infinite discrete spectrum, which we 
also construct (conjecturally almost all of it; see §9.8). We shall see that the (algebraic) 
situation is similar to the theta-correspondence, where the metaplectic representation is 
restricted to analogous subgroups. 

Let us here state the main results in a little more precise form, referring to sections 8 
and 9 for further notation and details. 

Typeset by AmS-TeX. 
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Theorem A (the branching law for 0(p,q) I 0(p,q') x 0{q"); see Theorem 7.1). // 
q" > 1 and g' + q" = q, then the twisted pull-back Q\ of the local conformal map $1 
between spheres and hyperboloids gives an explicit irreducible decomposition of the unitary 
representation w p,q when restricted to 0(p,q') x 0(q"): 

oo 

1=0 ' 2 

The representations appearing in the decompositions are in addition to usual spherical 
harmonics 7i (W) for compact orthogonal groups 0(q), also the representations 7r+\ for 
non-compact orthogonal groups 0(p,q). The latter ones may be thought of as discrete 
series representations on hyperboloids 

X(p, q) := {x = (cc', x") G W +q : \x'\ 2 - \x"\ 2 = 1} 

for A > or their analytic continuation for A < 0; they may be also thought of as cohomo- 
logically induced representations from characters of certain ^-stable parabolic subalgebras. 
The fact that they occur in this branching law gives a different proof of the unitarizability 
of these modules k v +\ for A > — 1, once we know w p,q is unitarizable (cf. Part I, Theo- 
rem 3.6.1). It might be interesting to remark that the unitarizability for A < (especially, 
A = — \ in our setting) does follow neither from a general unitarizability theorem on 
Zuckerman-Vogan's derived functor modules [31], nor from a general theory of harmonic 
analysis on semisimple symmetric spaces. 

Our intertwining operator (<E>i)* in Theorem A is derived from a conformal change 
of coordinate (see §6 for its explanation) and is explicitly written. Therefore, it makes 
sense to ask also about the relation of unitary inner products between the left-hand and 
the right-hand side in the branching formula. Here is an answer (see Theorem 8.5): We 
normalize the inner product || (see §8.3) such that for A > 0, 

11/11% = A||/||! 2(X(M)) , for any / G 

Theorem B (the Parseval-Plancherel formula for 0(p, q) I (9(p, q') x 0(q")). 

1) If we develop F G Ker Am as F = ^1^1^ according to the irreducible decompo- 
sition in Theorem A, then we have 

00 

11 77-11 2 _ Y^iipt 1 )^ 11 p( 2 )||2 

\\ r \\mP<« — / j \\ r l H^P.a' n II r l Hl 2 (S9"- 1 ) - 
1=0 +, ! + 

2) In particular, if q" > 3, then all of 7r^ + „» 1 are discrete series for the hyperboloid 
X(p, q') and the above formula amounts to 



11 77-112 _ST (1 ,q miF^ 1 )!! 2 II p( 2 )|l 2 

\\f \\mp<i - + ~o ~ 1 " l2 ( x (p,q'))" 1 IIl 2 (S9"-i)- 
1=0 

The formula may be also regarded as an explicit unitarization of the minimal represen- 
tation w p,q on the "hyperbolic space model" by means of the right side (for an abstract 
unitarization of w p,q , it suffices to choose a single pair (q', q")). We note that the formula 
was previously known in the case where (q', q") = (0, q) (namely, when each summand in 
the right side is finite dimensional) by Kostant, Binegar-Zierau by a different approach. 
The formula is new and seems to be particularly interesting even in the special case 
q" = 1, where the minimal representation w p,q splits into two irreducible summands when 
restricted to 0(p, q — 1) x 0(1). 

In Theorem 9.1, we consider a more general setting and prove: 
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Theorem C (discrete spectrum in the restriction 0(p 1 q) 1 0(p', q') x 0(p", q")). 
The twisted pull-back of the locally conformal diffeomorphism also constructs 

\eA'(p',q')nA'(q",p") XeA'(q',p')nA'(p",q") 

as a discrete spectrum in the branching law for the non-compact case. 

Even in the special case (p",q") = (0, 1), our branching formula includes a new and 
mysterious construction of the minimal representation on the hyperboloid as below (see 
Corollary 7.2.1): Let W p ' r be the set of if -finite vectors (K = 0{p) x 0(r)) of the kernel 
of the Yamabe operator 

Ker A x(p , r ) = {/ e C°°(X(p, r)) : A xfer) / = i(p + r - l)(p + r - 3)/}, 

on which the isometry group (3(p, r) and the Lie algebra of the conformal group 0(p, r+ 1) 
act. The following Proposition is a consequence of Theorem 7.2.2 by an elementary linear 
algebra. 

Proposition D. Let m > 3 be odd. There is a long exact sequence 

_> VF 1 '™" 1 -^4 VF 2,m " 2 VF 3 ' 771 " 3 y3 > ym -) 2 VF 777-1 ' 1 ^"i 1 

smc/i t/iat Ker </? p is isomorphic to (w p ' q )K for any (p, q) such that p + q = m + 1. 

We note that each representation space W p,q ~ 1 is realized on a different space X(p, q—1) 
whose isometry group 0(p,q — 1) varies according to p (1 < p < m). So, one may 
expect that only the intersections of adjacent groups can act (infinitesimally) on Ker(p p . 
Nevertheless, a larger group 0(p, q) can act on a suitable completion of Ker tp p , giving rise 
to another construction of the minimal representation on the hyperboloid X(p,q — 1) = 
0(p, q — l)/0(p — 1, q — 1) ! We note that Ker ip p is roughly half the kernel of the Yamabe 
operator on the hyperboloid (see §7.2 for details). 

We briefly indicate the contents of the paper: In section 4 we recall the relevant facts 
about discretely decomposable restrictions from [16] and [17], and apply the criteria to our 
present situation. In particular, we calculate the associated variety of vj p,q as well as its 
asymptotic if-support introduced by Kashiwara-Vergne. Theorem 4.2 and Corollary 4.3 
clarify the reason why we start with the subgroup G' = 0(p,q') x 0(q") (i.e. p" = 0). 
Section 5 contains the identification of the representations 7r^' 9 A and 7r^' 9 A of 0(p,q) in 
several ways, namely as: derived functor modules, Dolbeault cohomologies, eigenspaces 
on hyperboloids, and quotients or subrepresentations of parabolically induced modules. 
In section 6 we give the main construction of embedding conformally a direct product 
of hyperboloids into a product of spheres; this gives rise to a canonical intertwining op- 
erator between solutions to the so-called Yamabe equation, studied in connection with 
conformal differential geometry, on conformally related spaces. Applying this principle in 
section 7 we obtain the branching law in the case where one factor in G' is compact, and 
in particular when one factor is just 0(1). In this case we have Corollary 7.2.1, stating 
that w p ' q restricted to 0(p,q — 1) is the direct sum of two representations, realized in 
even respectively odd functions on the hyperboloid for 0(p, q — 1). Note here the analogy 
with the metaplectic representation. Also note here Theorem 7.2.2, which gives a myste- 
rious extention of w p,q by w p+1,q ~ 1 - both inside the space of solutions to the Yamabe 
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equation on the hyperboloid X(p,q — 1) = 0(p,q — l)/0(p — l,q — 1). We also point 
out that the representations ft+ 9 x for A = 0, — | are rather exceptional; they are unitary, 
but outside the usual "fair range" for derived functor modules, see the remarks in section 
8.2. Section 8 contains a calculation of spectra of intertwining operators and gives the 
explicit normalization of a Parseval-Plancherel formula for the branching law, and finally 
in section 9 we use certain Sobolev estimates to construct an infinite discrete spectrum 
when both factors in G' are non-compact. We also conjecture the form of the full discrete 
spectrum (true in the case of a compact factor). It should be interesting to calculate the 
full Parseval-Plancherel formula in the case of both discrete and continuous spectrum. 

The first author expresses his sincere gratitude to SDU - Odense University for the 
warm hospitality. 

§4. Criterion for discrete decomposable branching laws 

4.1. Our object of study is the discrete spectra of the branching law of the restriction 
vj p ' q with respect to a symmetric pair (G, G') = (0(p, q), 0(p', q') x 0(p", q"))- The aim 
of this section is to give a necessary and sufficient condition on p',q',p" and q" for the 
branching law to be discretely decomposable. 

We start with general notation. Let G be a linear reductive Lie group, and G' its sub- 
group which is reductive in G. We take a maximal compact subgroup K of G such that 
K' := K fl G' is also a maximal compact subgroup. Let go = to + Po be a Cartan decom- 
position, and g = t + p its complexification. Accordingly, we have a direct decomposition 
0* = |* + p* of the dual spaces. 

Let 7T G G. We say that the restriction tt\g> is G' -admissible if tt\g' splits into a direct 
Hilbert sum of irreducible unitary representations of G' with each multiplicity finite (see 
[15]). As an algebraic analogue of this notion, we say the underlying (jj, K)-module n K 
is discretely decomposable as a (q' , K') -module, if ttk is decomposed into an algebraic 
direct sum of irreducible (q' , i^')-modules (see [17]). We note that if the restriction tt\k' is 
^'-admissible, then the restriction h\q> is also G'-admissible ([15], Theorem 1.2) and the 
underlying (jj, K)-module ttk is discretely decomposable (see [17], Proposition 1.6). Here 
are criteria for K '-admissibility and discrete decomosability: 

Fact 4.1 (see [16], Theorem 2.9 for (1); [17], Corollary 3.4 for (2)). 

1) If AS k(tt) H Ad* (K)Ct') 1 - = 0, then n is K' -admissible and also G' -admissible. 

2) If ttk is discretely decomposable as a (g! ', K') -module, then pr ^ fl ' (V s (ttk)) C Af*,. 

Here, ASk(k) is the asymptotic cone of 

Supp K (7r) := {highest weight of r G K : [n\ Kt) : r] ^ 0} 

where K is the identity component of K, and (t') 1 - C t* is the annihilator of t'. Let 
Afp (c p*) be the nilpotent cone for p. V s (tvk) denotes the associated variety of tvk, which 
is an Ad* (Kc) -invariant closed subset of Af*. We write the projection pr p ^ p , : p* — > p'* 
dual to the inclusion p' p. 

4.2. Let us consider our setting where ix = w p ' q and (G,G') = (0(p,q),0(p',q') x 
0{p",q")). 

Theorem 4.2. Suppose p' + p" = p (> 2), q' + q" = q (> 2) and p + q G 2N. Then the 
following three conditions on p' ,q' ,p" ,q" are equivalent: 
i) vj p,q is K' -admissible. 
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ii) is discretely decomposable as a (g', K') -module. 

iii) min(j/, q', p" , q") = 0. 

The implication (i) =>- (ii) holds by a general theory as we explained ([17], Proposi- 
tion 1.6); (ii) =>- (iii) will be proved in §4.4, and (iii) =>- (i) in §4.5, by an explicit compu- 
tation of the asymptotic cone ASK(m p ' q ) and the associated variety V (ro^ g ) which are 
used in Fact 4.1. 

Remark. Analogous results to the equivalence (i) 44> (ii) in Theorem 4.2 were first proved 
in [17], Theorem 4.2 in the setting where (G,G') is any reductive symmetric pair and 
the representation is any A q (X) module in the sense of Zuckerman-Vogan, which may be 
regarded as "representations attached to elliptic orbits" . We note that our representations 
w p,q are supposed to be attached to nilpotent orbits. We refer [20], Conjecture A to 
relevant topics. 

4.3. The following corollary is a direct consequence of Theorem 4.2, which will be an 
algebraic background for the proof of the explicit branching law (Theorem 7.1). 

Corollary 4.3. Suppose that min(j/, q',p", q") = 0. 

1) The restriction of the unitary representation w p,<1 \g' is also G -admissible. 

2) The space of K' -finite vectors zu^f coincides with that of K- finite vectors . 

Proof. See [15], Theorem 1.2 for (1); and [17], Proposition 1.6 for (2). □ 

A geometric counterpart of Corollary 4.3 (2) is reflected as the removal of singularities 
of matrix coefficients for the discrete spectra in the analysis that we study in §6; namely, 
any analytic function defined on an open subset M + (see §6 for notation) of M which is a 
X'-finite vector of a discrete spectrum, extends analytically on M if p" = 0. The reason 
for this is not only the decay of matrix coefficients but a matching condition of the leading 
terms for t — > ±oo. This is not the case for mm(p' 7 q',p" , q") > (see §9). 

4.4. Proof of (ii) =>■ (iii) in Theorem 4.2. 

We identify p* with p via the Killing form, which is in turn identified with M(jp, g; C) 

by 

M(p,q;C)^p, X»(° *y 
Then the nilpotent cone A/" p * corresponds to the following variety. 
(4.4.1) {X e M (p, q; C) : both X l X and l XX are nilpotent matrices} 

We put 

M 0)0 (p, q; C) := {X G M(p, q; C) : X f X = O, l XX = O}. 

Then M 0) o(p, <?; C) \{0} is the unique A'; ~ 0(p, C) x 0(q, C)-orbit of dimension p + q — 3. 
The associated variety V g (w p ^ q ) of w p,q is of dimension p + q — 3, which follows easily from 
the -KT-type formula of w p ' q (see [23], Theorem 3.6.1). Thus, we have proved: 

Lemma 4.4. The associated variety V g (w p ^ q ) equals Mo ; o(f>, <?; C) . 

The projection pr p ^ p / : p* — > p'* is identified with the map 

pv p ^ p/ : M(p,q;C) ^ M(p',q';C)(BM(p",q";C), (J* ^j^(X u X 4 ). 
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Suppose p'p"q'q" ^ 0. If we take 

X := E 1}1 — E p ' + i jq / + i + v 7 — l-Ey+1,1 + \/— Ti?i,p'+i G M 0j0 (p, q; C), 

then pr p ^ p ,(X) = (£1,1,-^+1^+1). But E hl ^K(p>, q ') and -^p'+W+i &K(p»,q"y 
Thus, pr ^ fl ,(X) ^ A/"*/. It follows from Fact 4.1 (2) that ro^ 9 is not discrete decomposable 
as a (g', i^')-module. Hence (ii) => (iii) in Theorem 4.2 is proved. □ 

4.5. Proof of (iii) => (i) in Theorem 4.2. 

We take an orthogonal complementary subspace t ' of t' Q in 6 — °(p) + c, (<?)- Let 
be a Cartan subalgebra of to such that t ' := t§ H t ' is a maximal abelian subspace in tg. 
We choose a positive system A + (t, t c ) which is compatible with a positive system of the 
restricted root system E(t, t"). Then we can find a basis {/$ : 1 < i < [|] + [|]} on v 7 — Tto 
such that a positive root system of t is given by 

A+(M C ) ={/< =b : 1 < x < J < [|]> 

U{/<±/i:[f] + l<<<j<[f] + [f]} 
u({±/,:l<Z<[|]} (p:odd)) 
u({±/»:[|] + l<i<[| ] + [§]} («:odd)), 

and such that 

min(p',p") min(<j',q") 

(4-5.1) v=i(o*= E R /<+ E M %W 

3=1 3=1 

if we regard (to)* as a subspace of (t§)* by the Killing form. 

Suppose p'q'p"q" = 0. Without loss of generality we may and do assume p" = 0, 
namely, G' = 0{p, q') x 0{q") with + q" = q. 

Let us first consider the case p 7^ 2. Then the irreducible (9(p)-representation 7i a (lR p ) 
remains irreducible when restricted to SO(p). The corresponding highest weight is given 
by af\. It follows from the if-type formula of w p ' q (Theorem 3.6.1) that 

Snpp K (w^) = {ah + 6/ [f ]+1 : a, b E N, a + | = b + |}. 

Therefore we have proved: 

(4.5.2) AS K (w™)=R+(f 1 + f m+1 ). 

Then AS K (w p ' g ) n v /T T(t / )* = {0} from (4.5.1) and (4.5.2), which implies 

AS K (w™) n v /z TAd*(K)(^ ) ± = {0} 

because (t ')* meets any Ad*(if)-orbit through (t' ) ± . Therefore, the restriction w p,q \K> 
is i^'-admissible by Fact 4.1 (1). 

If p = 2, then w p ' q splits into two representations (see Remark 3.7.3), say w_^ q and 
w 2 ! q , when restricted to the connected component SOq(2, q). Likewise, H a (M. p ) is a direct 
sum of two one-dimensional representations when restricted to 50(2) if a > 1. Then we 
have 

AS K (^l q )=R+(±f 1 + f p+1 ). 

Applying Fact 4.1 (1) to the identity components (G , G' ) of groups (G, G 7 ), we conclude 
that the restriction w± q \ K > is K -admissible. Hence the restriction w p ' q \K' is also in- 
admissible. Thus, (iii) => (i) in Theorem 4.2 is proved. 
Now the proof of Theorem 4.2 is completed. □ 
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§5. Minimal elliptic representations of Q(p,q) 



5.1. In this section, we introduce a family of irreducible representations of G = 0(p, q), 
denoted by 7r+ 9 A , k p _1\, for A G A (p,q), in three different realizations. These representa- 
tions are supposed to be attached to minimal elliptic orbits, for A > in the sense of the 
Kirillov-Kostant orbit method. Here, we set 



(5.1.1) A (p,q) 



( {AgZ + e+2 . A > -1} (p>l, g ^0), 

{AeZ+£±2 : A> f -1} (p> 1,5 = 0), 

(p= l,g^0) or (p = 0), 

I (p=l, ? = 0). 



It seems natural to include the parameter A = 0, — \ in the definition of A (p, q) as above, 
although A = — \ is outside the weakly fair range parameter in the sense of Vogan [33]. 
Cohomologically induced representations for A = — \ and A = — 1 will be discussed in 
details in a subsequent paper. In particular, the case A = — 1 is of importance in another 
geometric construction of the minimal representation via Dolbeault cohomology groups 
(see Part I, Introduction, Theorem B (4)). 

5.2. Let MP ,q be the Euclidean space M. p+q equipped with the flat pseudo-Riemannian 
metric: 

gm.P,i = dv 2 H h dv p -i 2 - dv p 2 dv n+ i 2 . 

We define a hyperboloid by 

X(p,q) := {(x, y) G W ,q : \x\ 2 — \y\ 2 = 1}. 

We note X(p, 0) — and X(0,q) =0. If p = 1, then X(p,q) has two connected 

components. The group G acts transitively on X(p, q) with isotropy subgroup 0(p — 1, q) 
at 

(5.2.1) x° :=\1,0,--- ,0). 

Thus X(p, q) is realized as a homogeneous manifold: 

X(p,5)~0(p, 9 )/0(p-l,Q). 

We induce a pseudo-Riemannian metric gx(p,q) on X(p,q) from R p ' g (see [23], §3.2), and 
write A X (p, q ) for the Laplace-Beltrami operator on X(p,q). As in [23], Example 2.2, the 
Yamabe operator is given by 

(5-2.2) A XM = A X(M) --(p + q-l)(p + q-S). 

For A G C, we set 

C?(X(p, q)) := {/ G C~pr(p, 5)) : A x(p , g) / = (-A 2 + \{p + q - 2) 2 )/} 

(5-2.3) = {/ G C»°(X(p, q)) : A x(p , q) f = (-A 2 + 1)/}. 

7 



Furthermore, for e = ±, we write 



C?(X(p,q)) € := {/ G C?(X(p,q)) : f(-z) = ef(z), z G X(p,q)}. 
Then we have a direct sum decomposition 

(5.2.4) C?{X{p, q)) = C?(X(p, q)) + + C A °° (X(p, q))_ 

and each space is invariant under left translations of the isometry group G because G com- 
mutes with A X ( Pi q)- With the notation in §3.5, we note if q = 0, then C^°(X(p, O))^^!)*; 
is finite dimensional and isomorphic to the space of spherical harmonics: 

H k (W>) ~ C?(X(p,0)) = C?(X(p,0)) agn( _ 1)k (k:=X + E^l). 

5.3. Let G = 0(p, q) where p, q > 1 and let 9 be the Cartan involution corresponding to 
K = 0(p) x O(q). We extend a Cartan subalgebra of ! (given in §4.5) to that of g , 
denoted by f)o- If both p and q are odd, then dimf)^ = dimtg + 1; otherwise f)3 = t^- The 
complexification of P)q is denoted by f) c . 

We can take a basis {fi : 1 < i < [ E ^ 3 -]} of (f) c )* (see §4.5; by a little abuse of notation 
if both p and q are odd) such that the root systems of g is given by 

A( ,f) c ) ={±(/i±/i):l<i<J<[^]} 



u({±/,:l<Z<[£±^]} (p + ?:odd)^) 



Let {if,} C f) c be the dual basis for C (f) c )*. Set t := C#i (c t c C f) c ). Then the 
centralizer L of t in G is isomorphic to 5"0(2) x 0(p — 2,q). Let q = [ + u be a (9-stable 
parabolic subalgebra of g with nilpotent radical u given by, 

A(u, f) c ) := {/i ± fj : 2 < j < [E±l]} u ({±/x} (p + ? :odd) ) , 

and with a Levi part [ = [q <8> C given by 

[ = Lze(L) ~ o(2) + o(p-2,q). 

Any character of the Lie algebra lo (or any complex character of I) is determined by its 
restriction to \)q. So, we shall write for the character of the Lie algebra to whose 
restriction to h c is v G (f) c )*. With this notation, the character of L acting on A dimu u is 
written as C 2p ( u ) where 

(5.3.1) p(u) := (*±« - 1)A. 

The homogeneous manifold G/L carries a G-invariant complex structure with canonical 
bundle A top T*G/L ~Gx t C 2p ( u ). As an algebraic analogue of a Dolbeault cohomology 
of a G-equivariant holomorphic vector bundle over a complex manifold G/L, Zuckerman 

introduced the co ho mo logical parabolic induction 7Z J q = (j G N), which is a co- 

variant functor from the category of metaplectic ([, (L n K)~)-modules to that of (g, K)- 
modules. Here, L is a metaplectic covering of L defined by the character of L acting on 



A dimu u ~ < C 2 p( u ). In this paper, we follow the normalization in [32], Definition 6.20 which 
is different from the one in [30] by a 'p-shift'. 

The character Ca/ x of Iq lifts to a metaplectic (I, (L n -KT)~)-module if and only if 
A G Z + In particular, we can define (g, -KT)-modules ^(Ca/J for A G Aq(p, q). The 
Z(g) -infinitesimal character of ^(Ca/J is given by 

(A> £+3 _ 2 _ £+!_,_ ...^.[E+l^ft.). 

in the Harish-Chandra parametrization if it is non-zero. In the sense of Vogan [33], we 
have 

p + q 

C\f ± is in the good range A > — 2, 

Ca/j is in the weakly fair range A > 0. 

We note that ^(Ca/J = if j ^ p — 2 and if A G A (p : q). This follows from a general 
result in [31] for A > 0; and [14] for A = —\. 

5.4. For b G Z, we define an algebraic direct sum of K = 0(p) x 0((/)-modules by 

(5.4.1) E(K : 6) = S(0(p) x 0(g) : 6) := 7i m (lR p ) B 7i n (lR 9 ). 

m,n6N 
m — n>b 
m—n=b mod 2 

For A G A)(p, <?), we put 

(5.4.2) 6 = 6 (A,p,g) : =A-| + | + leZ, 

(5.4.3) e = e ( A ,p,g):=(-l) b . 

We define the line bundle £ n over O/L by the character nf\ of L (see §5.3). 
Here is a summary for different realizations of the representation ir+\: 

Fact 5.4. Let p, q G N > 1 

1) For any A G Ao(p, g), each of the following 5 conditions defines uniquely a (g, K)- 
module, which are mutually isomorphic. We shall denote it by (n+\)K- The (g, K)-module 
(n+\)K is non-zero and irreducible. 

i) A subrepresentation of the degenerate principal representation Indp ma x(e <S> C\) (see 
§3.7) with K-type Z(K :b). 

1) ' A quotient o/Indp ma x(e <E> C_a) with K-type "E(K : b). 

ii) A subrepresentation of C^ 3 (X (p, q))x with K-type "E(K : b). 

iii) The underlying (g, K)-module of the Dolbeault cohomology group 

H P q (G/L,£(\+e±il=2.))k- 

iii)' The Zuckerman- Vogan derived functor module lZq~ 2 (Ca/J • 

2) In the realization of (iii), if f G (7t+\)k, then there exists an analytic function a G 

/(wcosh^sinht) = a(u, 7])e~ (x+p)t (l + te~ 2t O(l)) as t -> oo 
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Here, we put p = p+q 2 ■ 

For details, we refer, for example, to [11] for (i) and (i)'; to [27] for (ii) and also for 
a relation with (i) (under some parity assumption on eigenspaces); to [15], §6 (see also 
[14]) for (iii)' <^> (ii); and to [34] for (iii) (in)'- The second statement follows from a 
general theory of the boundary value problem with regular singularities; or also follows 
from a classical asymptotic formula of hypergeometric functions (see (8.3.1)) in our specific 
setting. 

Remark. 1) By definition, (i) and (i)' make sense for p > 1 and q > 0; and others for 
p > 1 and q > 0. 

2) Each of the realization (i), (i)', (ii), and (iii) also gives a globalization of 7r^.' g A , namely, 
a continuous representation of G on a topological vector space. Because all of (tt 1 ^ q x )k 
(A G A (p, q)) are unitarizable we may and do take the globalization 7r^' a to be the unitary 
representation of G. 

3) If A > and A G Ao(p,q), then the realization (ii) of n+ q x gives a discrete series 
representation for X(p,q). Conversely, 

{n p + q x :\eA (p,q),\>0} 

exhausts the set of discrete series representations for X(p, q). 

If (p,q) = (1,0), then 0(p,q) ~ 0(1) and it is convenient to define representations of 
0(1) by 

(1 (A=-i), 
y (otherwise). 

As we defined 7r^' A in Fact 5.4, we can also define an irreducible unitary representation, 
denoted by ir^\, for A G A (q,p) such that the underlying (g, K)-module has the following 
K-type 

H m (R p ) Kl H n (R q ). 

m-n<-A+f-f -1 
m— n= — A+-§ — 2 — 1 mod 2 

Similarly to tt+ 9 a , the representations 7r^.' A are realized in function spaces on another 
hyperboloid 0{p,q) /0{p,q — 1). 

In order to understand the notation here, we remark: 

i) 7r^' A G 0(p,q) corresponds to the representation irV P x G 0(q,p) if we identify 0{p,q) 
with 0(g, p). 

ii) 7r^° A ~ H k (W), where = A - ^ and p > 1, A; G N. 

5.5. The case A = ±| is delicate, which happens when p + q G 2N + 1. 

First, we assume p + q £ 2N + 1. By using the equivalent realizations of ft+ q x in Fact 5.4 
and by the classification of the composition series of the most degenerate principal series 
representation Indp ma x(e <8> C\) (see [11]), we have non-splitting short exact sequences of 
(g, K)-modules: 

9+1 



(5.5.1) O-Ctt^Ok ->Ind? m «((-l) £= * L ®C_i) - 0, 

' 2 2 ' ' 2 



(5.5.2) 0^(<' 9 _Ok -+Ind£ max ((-1)^— ®C_i) 

-r, 2 2 ' 2 
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0. 



Because 7r^' g A (A G A (p, q)) is self-dual, the dual (g, X)-modules of (5.5.1) and (5.5.2) give 
the following non-splitting short exact sequences of ($j, K)-modules: 

(5.5.3) -^Indg^CC-l) 2 ^ 1 (8)Ci) ^(^'Sk -> 0, 

(5.5.4) 0^(7^)^ ^Ind£ max ((-1)T ®Ci) ->0. 

Next, we assume p + q G 2N. Then, ro p ' 9 is realized as a subrepresentation of some 
degenerate principal series (see [23], Lemma 3.7.2). More precisely, we have non-splitting 
short exact sequences of (jj, -ftT)-modules 

(5.5.5) -> -> Ind^ max ((-1)^ <g> C_i) -> ((tt™)* © -> 0, 

(5.5.6) -> © «' 9 !)k) -> Ind^ max ((-1)^ <g> d) -> ^ g -> 0, 

and an isomorphism of (g, K)-modules: 

(5.5.7) Ind^ max ((-1)T © C ) ~ « g ) x © «f )*. 

These results will be used in another realization of the unipotent representation w p,q , 
namely, as a submodule of the Dolbeault cohomology group in a subsequent paper (cf. 
Part 1, Introduction, Theorem B (4)). 

§6. CONFORMAL EMBEDDING OF THE HYPERBOLOID 

This section prepares the geometric setup which will be used in §7 and §9 for the 
branching problem of w p,q \c- Throughout this section, we shall use the following notation: 

P ' P " 

\ x \ 2 : = | x f + \ x "\ 2 = + 5^04) 2 5 for x := {x\ x") G R p ' +P " = R p , 

i=i j=i 

I II 

\y\ 2 ■■= \v'\ 2 + \y"? = i>9 2 + 1>;) 2 , for v : = rf) e Rq ' +q " = R9 - 

i=i 3=1 

6.1. We define two open subsets of W p+q by 

k£ +p "' 9 ' +9 " := {(x,y) = ((x',x"), (y',y")) G R p '+ p "> q '+ q " : \x'\ > \y'\}, 

K P -' +P "" ? ' + ' ? " := {(x,y) = ((x',x"), (y',y")) G R p '+ p "> q '+ q " : \ x '\ < \y>\}. 

1. n /, // /, // /, // 
Then the disjoint union R^ p ' q q UM^ ' q is open dense in R p+9 . Let us consider 

/. // /, // 

the intersection of R± ,q with the submanifolds given in §3.2: 

M C S C W' q . 
Then, we define two open subsets of M ~ x 5' 9-1 by 

(6.1.1) M ± :=Mnl p ± V ' ?V . 
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Likewise, we define two open subsets of the cone S by 

(6.1.2) E ± :=Ef]Ri +p "' q ' +q ". 
We notice that if (x, y) = ((x\ x"), (y', y")) £ H then 

\x I > \y | |x | < \y | 
because \x'\ 2 + \ x"\ 2 = \y'\ 2 + \ y"\ 2 . The following statement is immediate from definition: 

(6.1.3) S + = M + = p'q" = 0. 

(6.1.4) S_ = ^ M_ = ^ p'V = 0. 



6.2. We embed the direct product of hyperboloids 

X(p',q') x X{q\p") = {((*',</), {y\x")) : \xf - \yf = \y"\ 2 - \x"\ 2 = 1}. 
into S_|_ (c MJ ) ' q ) by the map 

(6.2.1) X{p',q') x X(q",p")^~ + , ((*', y'), (y",x")) ~ {x\ x\ y\ y"). 

The image is transversal to rays (see [23], §3.3 for definition) and the induced pseudo- 
Riemannian metric gx(p',q')xx(q",p") on X(p',q') x X(q",p") has signature {p' — l,q') + 
(p" , q" — 1) = {p — 1, q — 1). With the notation in §5.2, we have 

9X(p',q')xX(q", P ") = 9X(p',q') © ( ~ 9X(q" ,p") ) • 

We note that if p" = q' = 0, then X(p',q') x X{q" ,p") is diffeomorphic to S^ -1 x S 9-1 , 
and <7x(p' o)xX( g " o) is nothing but the pseudo-Riemannian metric ggp-ixS?- 1 °f signature 
(p- 1,5-1) (see [23], §3.3). 

By the same computation as in (3.4.1), we have the relationship among the Yamabe 
operators on hyperboloids (see also (5.2.2)) by 

(6.2.2) Ax(p',q')xX(q",p") = Ax(p',q') - ^X(q',p")- 

We denote by $i the composition of (6.2.1) and the projection $: S — > M (see [23], 
(3.2.4)), namely, 

(6.2.3) ^:X(p',q')xX(q",p")^M, ((x',y'),(y" ,x")) ~ ( ^ W ^ 



\x\ 



Lemma 6.2. 1) The map $1 : X(p\ q') x X(q" ,p") M is a diffeomorphism onto M + . 
The inverse map : M + — > -Y(p', q') x X(q" ,p") is given by the formula: 



(6.2.4) ((«',«"),(«',«")) 



(u',v') (v",u") 



vVl 2 - \v'\ 2 ' v^T 7 



2 _ U//I2 



2) $i is a conformal map with conformal factor \x\ 1 = \y\ 1 , where x = (x',x") G 
Rp'+p" and y = (y',y") e R q ' +q " . Namely, we have 

$1 (9SP~ 1 XS1~ 1 ) = T—^9X(p',q')xX(q",p")- 



\X\ 



Proof. The first statement is straightforward in light of the formula 

| It | — \V | = |t> | — ]lt ] > 

for (u, u) = ((«', u"), (v', v")) eM + C S?' 1 x 

The second statement is a special case of Lemma 3.3. □ 

6.3. Now, the conformal diffeomorphism $i : X(p',q') x X(q",p") M + establishes a 
bijection of the kernels of the Yamabe operators owing to Proposition 2.6: 
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Lemma 6.3. $^ gives a bijection from Ker onto Ker Ax(p',q')xx(q",p")- 
Here, the twisted pull-backs and (see Definition 2.3), namely, 

(6.3.1) 5* : C°°(M + ) -,r(l(p' ig ')xl( g '>")), 

(6.3.2) (^^W^xW)) - C°°(M + ), 
are given by the formulae 

(x\x") (y',y") 



(<E>*F)(x , y , y , x ) := (|x'| 2 + |xT)~^^ 



vV| 2 + |x"| 2 ' vW+W/ ' 



vVI 2 - M 2 ' vV'l 2 - KI 2 

respectively. We remark that ($^~ 1 )* = 

6.4. Similarly to §6.2, we consider another embedding 

(6.4.1) X(q>,p>) x X{p\q") S_, «y', x'), (*", I/")) - (*', x", y', y")- 

The composition of (6.4.1) and the projection $: S — > M is denoted by 

'(*',*") (y',y") 



(6.4.2) $ 2 : X (?',;/) x X(p",g") M, «y',x'), (x",y")) 



Fl 1 2/ 1 



Obviously, results analogous to Lemma 6.2 and Lemma 6.3 hold for <E> 2 . For example, here 
is a lemma parallel to Lemma 6.2: 

Lemma 6.4. The map $2 : X(q',p') x X(p",q") — > M_ is a conformal diffeomorphism 
onto M_. TTie inverse map $2 1 : — * X(q' ,p') x X(p" , q") is given by 



((u',u"),(v',v")) 



(«',«') (u",v") 



vV| 2 - |«'| 2 ' 



2 _ 



§7. Explicit branching formula (discrete decomposable case) 

If one of p',q',p" or q" is zero, then the restriction w p,<1 \g' is decomposed discretely 
into irreducible representations of G' = 0(p', q') x 0{p" , q") as we saw in §4. In this case, 
we can determine the branching laws of w p,q \G' as follows: 

Theorem 7.1. Let p + q G 2N. 7/ q" > 1 and g' + q" = q, then we have an irreducible 
decomposition of the unitary representation vj p,q when restricted to 0(p, q') x 0(q"): 



00 

(7.1.1) ^Yf^'^H 1 ^''). 



z=o 
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We shall prove Theorem 7.1 in §7.5 after we prepare an algebraic lemma in §7.3 and a 
geometric lemma in §7.4. 

Remark. The formula in Theorem 7.1 is nothing but a K-type formula (see Theorem 3.6.1) 
when q' = 0. 

7.2. The branching law (7.1.1) is an infinite direct sum for q" > 1. This subsection treats 
the case q" = 1, which is particularly interesting, because the branching formula consists 
of only two irreducible representations (we recall ^(R 1 ) 7^ if and only if I = 0, 1). For 
simplicity, we shall assume q > 3 in §7.2. 

It follows from Theorem 7.1 with q' = 1 that 

(7.2.1) ~ (tt^ 1 Si)© (ttJV 1 B sgn) . 

This means that can be realized in a subspace of CT(X(p 7 q — 1)), namely, the kernel 

2 

of the Yamabe operator A X ( p , q -i) (see (5.2.3)). 

More precisely, according to the direct sum decomposition (see (5.2.4)), we have 

(7.2.2) Ker A x(p , H) = CT (X(p, ? - 1)) = C?°(X(p, g - 1))+ + C£(X(p, q - 1))_. 
We recall that the central element ~I p + q £ G acts on ot^' 17 with scalar 5, where 

(7.2.3) 5:=(-l)^. 

In view of the composition series of eigenspaces on the hyperboloid (see [27] for the 
case 5 = +; similar for 5 = — ), we have non-splitting exact sequences of Harish-Chandra 
modules of 0(p, q — 1): 

(7.2.4) ^^J) K ^ (c?{X(p,q-l)) s ) -(tt™- 1 )* - 0, 

2 \ 2 / K '2 

(7.2.5) ^(Trfr 1 )^ -(ci°(X(p, q - l))s) ^ q ~l)K - 0, 

"T' 2 V 2 / if i 2 

Here is a realization of U7 P ' 9 in a subspace of the kernel of the Yamabe operator on the 
hyperboloid X(p, q— 1), on which 0(p, q) acts as meromorphic conformal transformations. 

Corollary 7.2.1. Let W± be the unique non-trivial subrepresentation of 0(p,q — 1) in 
(Ker Ax( p , g -i))± = C^(X(p, q— 1))±. Each of the underlying ($j, K)-module is infinites- 

imally unitarizable, and we denote the resulting unitary representation by W±. Then, the 
irreducible unitary representation zu p ' q of 0(p, q) is realized on the direct sum W + + W-. 

We note that WT S c L 2 (X(p, q - 1)) and W s <£ L 2 (X(p, q - 1)) where 5 = (-l)** 2 . 

It is interesting to note that the Laplacian A X ( P:q -i) acts on a discrete series 7r+^ _1 f° r 
the hyperboloid X(p, q — 1) as a scalar — A 2 + \ {p + q — 3) 2 , which attains the maximum 
when A = \ if p + (q - 1) e 2N + 1. 

Taking the direct sum of two exact sequences (7.2.4) and (7.2.5), we have the following: 
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Theorem 7.2.2. There is a non-split exact sequence of Harish- Chandra modules forO(p— 
1,?). 

(7.2.6) - (m™) K - (Ker & x(p , q - X) ) K - {^^k - 0. 

It is a mysterious phenomenon in (7.2.6) that w p ' q extends to a representation of 0(p, q) 
and (w p+1,<1 ~ 1 )k to that of 0(p + l,q — 1). So, different real forms of 0(p + q,C) act 
on subquotients of the kernel of the Yamabe operator on the hyperboloid X(p, q — 1) = 
0(p,q-l)/0(p-l,q-l) ! _ 

Here, we remark that Ker A X ( P:q -i) H L 2 (X(p, q — 1)) 7^ {0} if and only if p + g G 2Z, 
by the classification of discrete series for the hyperboloid X(p, q — 1) for p > 1. 

7.3. By Theorem 4.2, the restriction w p,q \K' is -KT'-admissible, where If' ~ 0(p) xO(q') x 
0(q"). Let us first find the if'-structure of w p,q . We recall a classical branching law with 
respect to (0(q),0tf) x 0(g")) (q = q' + q", q' > 1, q" > 1): 

(7.3.1) H n (R q ) ~ ft fc (M 9 ') K^(M g "). 

A;,Z6N 

Then we have isomorphisms as if'-modules: 

7Y m (lR p ) K7i n (lR 9 )| (p) x0 (g/) x o(g») 

m,rt6N 
m— n=b 
m—n=b mod 2 

- n m (R p )mn k (R q ')^n l (R q ") 

m,n6N A;,ZeN 
m — n=b k+l<n 
m—n=b mod 2 k+l=n mod 2 

- r(r)iK fc (R q ' ) b w 1 (m 9 " ) 

ZEN m,fc6N 
m — b>k+l 
m — b=k+l mod 2 

~0S(O(p) xO(g') :b + l)®H l (R q ") 

In view of Theorem 3.6.1, we have proved: 

Lemma 7.3. We have an isomorphism of K' -modules: 

(7.3.2) < 9 ^®S(0(p) x O(q') : + I) ® H l (R q " ) 

7.4. By Theorem 4.2 and [17], Lemma 1.3, the underlying (g, iiT)-module is decom- 
posed into an algebraic direct sum of irreducible ($j', K')-modules: 

(7.4.1) ~0m T r~0 m Tl , T2 Ti Sr 2) 

T Tl,T 2 

where m T G N and r runs over irreducible ($j', K')-modules or equivalently, T\ runs over 
irreducible (q[ 7 K()-modules (with obvious notation for G[ := 0(p,q')) and T2 runs over ir- 
reducible 0((/")-modules. It follows from Lemma 7.3 that for each / there exists a (fl^, K[)- 
module W\ which is a direct sum of irreducible (g' l7 -KT^-module such that W\ is isomorphic 
to E(0(p) x O(q') : + /) as K( -modules. Let us prove that Wi is in fact irreducible as 
a (qi, K{)-module. 

15 



Lemma 7.4. W\ is realized in a subspace of (X (p, </)) A = / + \ — 1. 

Proof. In our conformal construction of w p,q in §3, we recall that each K-finite vector of 
w p,q is an analytic function satisfying the Yamabe equation on M ~ S^ -1 x 5' 9-1 . By 
using the conformal diffeomorphism $i : -Y(p, q') x S q _1 — ■> M+, an open dense subset of 
M (see (6.1.4) with p" = 0), we can realize W\ x H, l (R q ) in the space of smooth functions 
on X(p,q') x S q _1 satisfying the Yamabe equation by the following diagram: 

A(M) C C°°(M) C°°(M+) ^C°°(X(p,q') x Si"' 1 ) 

dense restriction <3?^ 

u u u u 

WiBW'CR^CGT™ C Ker A M ^ Ker A M+ £ Ker A x(m , )xS ,»_, 

dense restriction <£>:[ ' 

Because ^x(p q^xSi"- 1 = ^x(p',q') — A s ,»_i (see (6.2.2)) acts on WiM7i l (M q ) as 0, and 
because A 5 ,»_i acts on 7i l (M, q ) as a scalar | — (/ + ^ — l) 2 (see (3.5.1)), we conclude 
that Ax(p', q ') acts on Wz as the same scalar. Hence, Lemma is proved. □ 

7.5. Let us complete the proof of Theorem 7.1. It follows from Lemma 7.4 together with 
the i^-structure of W\ in §7.3 that Wi is irreducible and isomorphic to (n p,q qI/ _ 2 )k' as 

(fli, KQ-module (see Fact 5.4). Therefore, we have an isomorphism of (g', X')-modules 



~ 0(7r p ' 9 ' ^ _ ) K , B ?^(R 9 ") (algebraic direct sum). 

Taking the closure in the Hilbert space, we have (7.1.1). Hence Theorem 7.1 is proved. □ 

7.6. So far, we have not used the irreducibility of vj p ' q in the branching law. Although 
the irreducibility of w p,q (p, q) ^ (2, 2) is known [2], we can give a new and simple proof 
for it, as an application of the branching formulae in Theorem 7.1. 

Theorem 7.6. Let p, q > 2, p + q e 2Z and (p, q) ^ (2, 2). Then, w p,q is an irreducible 
representation of 0(p, q) . 

Proof. Suppose W ^ {0} is a closed invariant subspace of the unitary representation 
(w p ' q , V p ' q ). We want to prove W = V p ' q . 

Without loss of generality (p and q play a symmetric role), we may assume q > 3, 

oo 



and fix q' > 1, q" > 2 such that q' + q" = q. We write V p > q = ^ V\ according to the 

1=0 

irreducible decomposition (7.1.1) of G' = G[ x G' 2 := 0(p,q') x 0{q"). 

Because is non-zero, W contains a -ftT-type of the form H, a (W p ) IE 7i & (M' ? ), which 
we fix once for all. In view of the branching formula (7.3.1), T-l b (M. q ) contains a non-zero 
0(q")-&xed vector. That is, 



V Q := V p ' qQ ' 2 D W G * D ft a (M p ) ®H b (M. q f' 2 ^ {0} 

Because = 0(p, </') acts on Vo by 7r p ' q qll (see Theorem 7.1) which is irreducible, and 

because Vb D IV 2 is stable under the action of G[, we conclude that Vo = W 2 . Hence, 
we have proved 

(7.6.1) W D Vq. 
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If W 7^ VP'i, the orthogonal complement W 1 - D V by the same argument. This would 
contradict to W n W 1 - = {0}. Therefore, W must coincides with VP'i. Hence, m p > q is 
irreducible. □ 



§8. Inner product on w p ' q and the Parseval-Plancherel formula 

In this section, we prove the Parseval-Plancherel type formula for our discrete decom- 
posable branching law given in Theorem 7.1. Our main result in this section is Theorem 
8.5. 

8.1. The subsections §8.1 ~ §8.3 review some results and notation from [21] without 
proof. We set 



A+(p, q) 
A+-(A) 
A++(A) 



A (p, g)n{AeK:A>0}, 

{(A', A") e A+(p', q') x A+(q",p") : A' — A" — A — 1 G 2N}, 
{(A', A") G A+(p', q') X A+(p", q") : A - A' - A" - 1 G 2N}. 



We recall the Jacobi function 



(A 
fix 



which satisfies 



™(t) := ( V + A ; +1 - A , A/ + A ; +1 + A ; A" + l; - t ) , 

v£" ,V) (o) = i, 

t(l-t)^ + (c-(a + b+ l)t)j t - a&J <p iX (t) = 



with 



a 



A' + A" + l-A X' + X" + l + X x „ 1 

6:= , c:=A +1. 



2 7 2 

We set meromorphic functions of three variables A, A' and A" by 



V 



(A',A") ._ 



(r(A" + i)) 2 r( 



-) 



+- A 



2A r( A+A 2 +A+1 ) r( A +A 2 - A+1 ) 
( v,a») ._ (r(A" + i)) 2 r( -^ / -^+ 1 )r( ^ / -^+ 1 ) 



++,A 



2A r(- 



-A'+A" + A+^ tvA'+A"+A+1 



-) r(: 



) 



Then we have 



/•OO 

/ \<p\x' ,A ' ) (t)| 2 (cosht) 2A '+ 1 (sinht) 2A "+ 1 = V^'P for (A', A") G A+_(A), 
Jo 

i: 



^■x' ,X '\^)\' 2 (cosey 2A +i (sin#) 2A +1 d# = ; for (A', A") G A++(A) 



(A',A") 



8.2. We put 



(8.2.1) M = M (A, A', A") := (-1) 



v-y-.-i r( A/+A, ;- A+1 )r(A + i) 



r( A, - A, ;+ A+1 )r(A" + i) 
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Lemma 8.2 (triangular relation of the Jacobi function), if cot 6 = cosht (0 < 9 < f , < 
t), then 

<p& X "\i9) = M(cosh^+^''+VSf > X '\t). 
Furthermore, we have 

(8.2.2) ^'V++;y = M 2 \V^ r) . 

8.3. Unitarizaition of -k™ x (A G A (p,q)). 

In this subsection, we give an explicit unitary inner product on 7r^' g A for A G Aq(j>, q). 
In view of Fact 5.4, the (g, -ftT)-module (tt^\)k is realized in C^(X(p, q))K with X-types 
E(K : b) where 6 = A— f + f + 1. Suppose / G C^°(X(p, g))* belongs to the if-type 
H m {W) B H n (M Q ) G S(K : 6). Then / is of the form 

(8.3.1) /(wcosht,?7sinht) = / im (w)/i n (?7)(cosht) m (sinht) n ^ +f _1 ' m+f _1) (t), 
where h m G 7i m (R p ), h n G 7i n (lR 9 ), w G S p -\ n G S 9 " 1 , and t > 0. We put 

(8.3.2) ll/ll^™ := ll^m|||2(5 P -i) ||^n||l2(59-i) <M+_,A 2 2 \ 

This norm defines an inner product on (ir+ q x )K (A G A (p, (/)) and one defines an ir- 
reducible unitary representation of G = 0(p,q) on its Hilbert completion. There is an 
obvious inner product for A > 0, because (tt+\)k C L 2 (X(p, q)). The relation between 
our norm || and L 2 -norm || \\L 2 (x( P ,q)) is given by 

(8-3.3) ||/||^ = A||/||| 2(x(p)?)) , for any / G (**« X ) K , 

if A > 0. For A G A (p, q) such that A < (this can happen if A = 0, — |), 7r p ' 9 A is also 
unitarizable by this inner product ( , ) 7r p ' q given in (8.3.2), which is still positive definite. 

Remark 8.3 (Unitarity). As we explained, all of (tt^ \)k are unitarizable for A G Ao(p, q). 
We explain four different approaches: 

1) If A > 0, then (n^ q x ) K is unitarizable because of the realization in L 2 {X{p, q)). 

2) If A > 0, then (n^ q x )K is unitarizable because of the realization of Zuckerman-Vogan's 

derived functor module 7Zq~ 2 (C\) with the parameter A in the weakly fair range [31]. 
We note that the case A = — ^ is not treated in the above two methods. However, 

3) Use the classification of unitarizable subquotients of Indp ma x(e ® Ca) given in [11], §3. 
At last, here is a new proof of the unitarizability of (n 1 ^ q x )k for all A G Aq(p, q). The 

idea is to use our branching formula Theorem 7.1, for which the proof does not use the 
unitarizability of (tt^^k- 

4) All of (7r^' 9 A )x are unitarizable because they appear as discrete spectra in the branching 
law of a unitary representation w p,q+c of a larger group 0(p, q + c) to 0(p, q) x O(c) 
for some c > (see Theorem 7.1). For this purpose, c < 3 will do. 

8.4. We notice that the map $i : X(p, q') x S q " ~ x -> M + (6.2.3) (in the case p" = 0) is 
given by 

((cj cosht, rj' sinht), rj") i— > (u>, (rj' cos 6>, rj" sin 6*)), 
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where 9 and t satisfy cot6> = cosht. Suppose / G C°°(X(p, q') x S q " 1 ) belongs to 

as an O(p') x 0(g")-module, and furthermore to H m (MP) R H n (R q ' ) as an 0(p) x 0{q')- 
module in the first factor. Then / is of the form: 

Z(( W cosht,r/sinht),7/') = h m {u) h k ( V ') /^(^(coshtnsinht) k ^J~''^ ~ 1} (t) 

where h m G H m (W), h k G H k (R. q '), h t G H l (R q "), u G S p -\r/ G S q '-\r]" G S 9 "" 1 , and 
t > 0. 

Lemma 8.4. The twisted pull-back i^ 1 )* : C°°(X{p,q[) x S^"" 1 ) -> C°°(M + ) fsee 
(6.3.2) for definition) is given by the formula: 

(8.4.1) (($r 1 )*/)(^,(Vcos^7 ? // sin^)) 

= M- 1 h m {oo) h k {rf) httf') (cos9) k (sm9) 1 ^ff^^ 

Proof. In view of the definition, we have 

(($^" 1 )*/)(cu, (// cos 0,77" sin 0)) = (cosht) 2 ^ - ^/((cucosht, r/sinht), 77"). 
Then (8.4.1) follows from Lemma 8.2 and from m + | = n + |. □ 

8.5. Let G" = 0(p, </') x 0(q"). We define an inner product on the irreducible constituent 
7T+ a ^7T_ a' in the branching law G7 P '«| G ' (see Theorem 7.1) for each A = / + ^-1(/gN) 
by' 

11/11 : = ll/l|l w P.«' 11/2 || £,3(5*" -1) 

for f = hf 2 with h G k p + % and f 2 G tt^' ~ 

Because we use an explicit map to prove the branching law, the generalized Parseval- 
Plancherel formula makes sense. 

Theorem 8.5. 1) // we develop F G KerAM as F = Y^T accor ding to the 

irreducible decomposition 

00 

(8.5.1) (i^: w p ' q \ ( P , q ')xO(q") ^ Y^^+sl-! K ^( Rg ") 

OO 

(8-5.2) \\F\\i P , q =J2\\FU\\l P:Q , 

1=0 +.'+V 
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2) In particular, if q" > 3, then all °/ 7r ^ + „" 1 are discrete series for the hyperboloid 
X(p, q') and 

°° q" 

(8.5.3) ll^llwp-g = y^(^ + y - 1 ) \\ F i lll 2 (x(p,q'))H F i IIl 2 (5>j"-i)- 

1=0 



Remark. The formula (8.5.3) coincides with the Kostant-Binegar-Zierau formula (see §3) 
in the special case where q' = (namely, where G' is compact). 

Proof. We write A := I + ^ - 1, A' := m + \ - 1 = n + § - 1 and A" := k + ^ - 1. If F 
is of the form of the right side of (8.4.1), then 



V^l) F \\^ q i®H l (R«") M2 ll^m|||2( 5P -i) ||M|^ 2 ( 59 '-i) IIM^gV'-i) W- 



+-,A 



I _P II 2 „ ill, ||2 ill, 1 1 2 ill, ||2 \'T/( A "' A ) 

1 lltU ^' y ||'lm|| L 2( S p-l) ||'ifc|l L 2( S q'-l) IRH^^g"-!) K + + ,A' 

= 1 



because of (8.2.1). Hence the first statement is proved. The second statement follows from 
(8.3.2). □ 

§9. Construction of discrete spectra in the branching laws 

9.1. In §7, we determined explicitly the branching law w p,q \c of the minimal unipotent 
representation w p,q G G where G' = 0(p,q') x 0(q"). The resulting branching law has 
no continuous spectrum (see Theorem 4.2 and Theorem 7.1). In this section, we treat 
a more general case where continuous spectrum may appear, namely, the branching law 
with respect to the semisimple symmetric pair 

(G,G>) = (0(p,q),0(p>,q') x 0(p",g")), 

where p' + p" = p (> 2), q' + q" = q (> 2), p + q G 2N, and (p,q) ^ (2,2). 

We shall construct explicitly discrete spectra by using the conformal geometry. 
Retain the notation in §5.1. We set 

(9.1.1) A'(p, q) := A {p, q) n {A G R : A > 1}. 

Theorem 9.1. The restriction of the unitary representation w p,<1 \g' contains 

\eA'(p',q')nA'(q",p") XeA'(q',p')nA'(p",q") 

It lilt / / 

as a discrete summand. Here, 7r^ ' q M ir_ ' q G G' is the outer tensor product of 71^ ,q G 
6{p^q') and n^'f" G 0{q^p"). 

We have already established the full branching law zu p ' q \c if p'q'p"q" = 0. Thus, the 
main part of this section will be devoted to the proof of Theorem 9.1 when p'q'p" q" ^ 0. 
We shall give some remarks of Theorem 9.1 at the end of this subsection. 
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9.2. Let K' = 0(p') x O(q') x 0(p") x 0(q"). We realize the unitary representation w p ' q 
on the Hilbert space V p > q . Here we recall the notation of §3.9, briefly as follows: 



yp,g = Ker(D p - D q ) C V = Dom(D p ) n Dom(D,) ( C L 2 (M)) 

closed ' dense 

M = S p ~ 1 xS q - 1 D M + UM_. 

dense 

Different from Corollary 4.3 in the discretely decomposable case (p'q'p"q" = 0), a .KT'-finite 
vector of a G'-irreducible summand (i.e. a discrete spectrum) in w p,q \c is not necessarily 
a real analytic function on M if p'q'p"q" ^ in our conformal construction of zu p ' q . With 

this in mind, we extend ($^~ 1 )*/ G C°°(M+) (see §6 for notation) to a function, denoted 
by T+f, on S^ 1 x S^ 1 as 

(9.2.1) T+f := { on M +' 

[O on M\M + . 

Here is a key lemma: 

Lemma 9.2. Suppose X' G A (p',q') and X" G A (q",p"). Let f G C°°(X(p',q') x 
X(q",p")) be a K' -finite function which belongs to K+'y Kl k p _ y, (see Fact 5.4 (1) (H))- 

1) IfX'>\ and X" > \, then T+f G V. 

2) If X' > 1 and X" > 1, toen F(T+/) G L 2 (M) and yy'( T +/) e L X (M) /or any smooi/i 
vector field Y, Y' on M . 

3) // A' = A" > 1 tfien T+f G F^. 

Before proving Lemma 9.2, we first show that Lemma 9.2 implies Theorem 9.1. In fact, 
Lemma 9.2 (3) constructs a non-zero (q', K')-homomorphism 

T+:{4 : i) K i®{^/) K i^^ q , 

for A G A'{p',q') n A' (<f ',&'). Then T + is injective because (tt+^V; ^ {^-,\")k' 2 is 
irreducible. Since ro p ' g is a unitary representation of G, T+ extends to an isometry of 
unitary representations of G'\ 



by taking the closure with respect to the inner product induced from w p,q . This proves 
Theorem 9.1 for the irreducible representation n+'l Kl 7r p j^ 7 that appears in the first 
summand. The second summand is constructed similarly by using the conformal diffeo- 
morphism (see Lemma 6.4) 

<f> 2 :X(q',p')xX(p",q")^M_ (c M). 

Hence, the proof of Theorem 9.1 is completed by assuming Lemma 9.2. 
9.3. 

Remark 9.3. There exist A' G A (p', q') and A" G A (q",p") satisfying A' = A" > 1 if and 
only if p' + q' = p" + q" mod 2 and p' > 2, q" > 2. This implies p + q G 2N, p > 2 and 
q > 2, and F p ' g is non-zero (see §3). Of course, Lemma 9.2 (3) also implies V p,q ^ {0}. 
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9.4. It follows from Lemma 3.8.1 that the first statement of Lemma 9.2 is proved if 
Y(T + ) E L 2_e for any e > and for any smooth vector field Y on M. Therefore, both of 
(1) and (2) of Lemma 9.2 are proved by studying the asymptotic behaviour of T + f near 
the boundary of M + in M. This asymptotic estimate is studied below. 

Any X '-finite vector / is a finite linear combination of the form /1/2, where 

/1 G C°°{X{p', q')) is an 0(p') x 0(g')-finite vector that belongs to tt^,', 
h e C°°(X(q",p")) is an 0{p") x 0{q") -finite vector that belongs to n^'f". 

In order to prove Lemma 9.2, we may and do assume / is of the form /1/2. Then we have 

(9.4.1) (r + (/!/ 2 ))(«,t;) 



O + O — 4 

= (\u>\ 2 ^" 2 ^- — 



- r ^, , (u',v') \ f {u",v") 

v VWW^W J \ vV'l 2 - w? 



Here, we have used the following notation: 



r v (r > 0), 
+ "~ 1 (r<0). 



rl := 



9.5. In order to analyze the asymptotic behaviour of T+(/i/2) (see (9.4.1)) near the 
boundary of M + , we consider a change of variables on S'p -1 x S' 9-1 by the surjective map 

(S 1 x ^p'" 1 x Sp"- 1 ) x (S 1 x S 11 '- 1 x S 9 "" 1 ) -> S^" 1 x S q ~\ 
(e*V>'0,(e^r,W) ->(«, «) 

defined by 

(9.5.1) (it, i>) = (u',u" ,v',v") := (u/cos#,u/' sin 0,77' cosy?, 77" sin y?). 

Because |i/| 2 — |i/| 2 = |u/cos6>| 2 — |t7' cos </?| 2 = |cos6>| 2 — | cos v?| 2 , M± defined in (6.1.1) 
is rewritten as 

M_|_ = {(u/ cos 6, ui" sin 9, 7/ cos <p, rj" simp) : | cos 9\ > | cos <p\} 

M_ = {(u/ cos 9, u" sin 9, 7/ cos </?, 77" siny) : | cos 9\ < | cos 99 1} 

Here is an elementary computation corresponding to the change of variables (9.5.1): 
Lemma 9.5. 1) The volume element du dv on S p ~ x x S q ~ x is given by 

du dv = Icos^'- 1 I sin^l^"- 1 Icosv?! 9 '" 1 Isinv?! 9 "" 1 d9 dip du' du" dn' drj" , 

where du' is the volume element on S p _1 and so on. 

2) Any smooth vector field on >S P_1 x .S 9-1 is a linear combination of 

1 j 'r 'r 'r 3 3 



cos 9 ' sin 9 ' cos cp ' sin ip ' dip ' 89 
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whose coefficients are smooth functions of (u/ , u" , rj' , rj" , 9 , cp) . Here, X',X",Y' and Y" 
are smooth vector fields on S p _1 , S p _1 , S q _1 and S q ~ x , respectively. 

Fact 5.4 (2) describes the asymptotic behaviour of K-finite functions that belong to 
discrete series representations for a hyperboloid. Applying it to f\ with respect to the 
coordinate (9.5.1), we find a x G C 00 ^'- 1 x S^'-i) an d ^ G C°°(R) such that 

2X'+p'+q'-2 



, (u',v') \ . . /cos 2 9 + cos 2 (p\ 4 , / / cos 2 9 — cos 2 <p 



v /|^ 7 |2 _ / v ' ' ; Vcos 2 ^ - COS 2 (/7/ + I y cos 2 (9 + cos 2 y 

Likewise, there exist a 2 G C 00 ^"" 1 x Sp"- 1 ) and /i 2 G C°°(M) such that 



2\"+p" +q "~2 



i (v",u") \ ., ... ( sin 2 9 + sin 2 (p \ 4 / / cos 2 9 — cos 2 ip 



^/|u"|2 - | u "|2 J w ' Vcos 2 6'-cos 2 (/7/ + I y sin 2 (9 + sin 2 </? 

We treat the boundary dM + of M + C M locally in the following three cases: 
Case 1) cos 2 9 — cos 2 (p = 0, (cos6>, cos (p) ^ (0, 0), (sin6>, sin</?) ^ (0, 0). 
Case 2) cos 9 = cos </? = 0. 
Case 3) sin6> = sirup = 0. 

We note that Case (2) or (3) happens only when p'q'p"q" ^ 0. 

9.6. (Case 1): In this subsection, we consider a generic part of the boundary dM + 
corresponding to Case 1. In a local coordinate (u/,u/', rf , 77", 9, (p) G5 ,p_1 x5 ,p_1 x 
S g'-i x g g "-i x T + {f 1 f 2 )){u,v) is written as 

p + q-4 . 2A / +p'+<;'-2 . 2\"+p"+q" -2 \'+\" 

A (cos 9 - cos <p) + 4 + 4 + 4 = A (cos 6> - cos tp) + 2 . 

1 

Here, A is a smooth function of variables u', u>", rj', n" ', (cos 2 6> — cos 2 </?) + . Therefore, by 
using Lemma 9.5, we have: 

A' + A">-1 T+fezlc, 

A' + A" > 1 Y(T+f) G L 2 " e for any e>0,F6 £(M), 

A' + A">2 Fiy 2 (T + /) G foran y y 1 ,y 2 GX(M), 

in a neighbourhood of the boundary point of M + for Case (1). 

9.7. (Case 2): In this subsection we consider a neighbourhood of (u/, u/', 77', 7/", 0, <p) 
satisfying the condition of Case 2. We take a polar coordinate for (cos6>, cos</?) °= (0, 0) as 

cos6> = rcosV>, 

cos ip = r sin ip. 

The composition to (9.5.1) yields a new coordinate on S^ -1 x S q ~ 1 given by 

(■u,i>) = (u', u" , v' , v") = (u'r cos ip, u" yfl — r 2 cos 2 ip, rj'r sin ip, rj" \Jl — r 2 sin 2 -0) , 

where w' G w" G S p ' ' ~ x , 77' G S 9 '" 1 , 77" G r > 0, and ip G M. Then our 

interest is in a neighbourhood of r = 0. In this coordinate, we have 

M+ = {r ^ 0,cos2t/> > 0} 

The Jacobian matrix of the transform (9, <p) — > (r, is given by 





9r \ 






d V 1 










V 9f 







— cos t/> sin ^ — sin t/> sin </? 
^sin ^ sin 9 ^cost/* siny? 
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Lemma 9.7. 1) The standard measure on S p 1 x S q 1 is locally represented as 

smooth function of (r, ip, u/, r\ , u" , n") x r p +q _1 drdipduj' dn' du" dn" . 
2) j4m/ smooth vector field on ,S P_1 x S" 9-1 near r = is a linear combination of 

1 y> x » _J_ Y > Y" 9 1 9 

r cos ^ r sin -0 ar r ch/* 

whose coefficients are smooth functions of (u' ,u>" ,rj' ,rj" ,r,ip). Here, X',X",Y' and Y" 
are smooth vector fields on S p _1 , S p _1 , S q _1 and S q -1 , respectively. 

By noting the relations 

cos 6* + cos 99 1 sin + sin 9 2 — r 

cos 2 6* — cos 2 99 cos 2ip ' sin 2 <p — sin 2 9 r 2 cos 2ip ' 

T + (/i/2)(n, n) is locally written as 

p + q-4 2A / +p / + q / -2 2A"+p" + < 3 "-2 

S(r 2 cos2^) ^(cos2^) + 4 (r 2 cos2^) + 4 

2\"-p'- q ' + 2 a' + a" 

=B r + 2 (cos 2V0+ 2 , 

i 

where S is a smooth function of variables u', u", rj', rj" , r+, (cos2?/j)^_. Therefore, by using 
Lemma 9.7, we have: 

2A">-1, A' + A">-1 =► T + /GL 2 oc , 

2A" > 0, A' + A" > 1 =>• Y(T+f) G L 2 ~ e for any e>0,Fe X(M), 

2A" >2-p'- q',X' + A" > 2 Y 1 Y 2 (T+f) G L\ oc for any Y u Y 2 G X(M), 

in a neighbourhood of the boundary <9M + for Case (2). 

The asymptotic estimate for Case (3) parallels to that of Case (2). 

, , 2X"-p'-q'+2 

Remark 9.7. Assume A' = A" G Z + p ^ . Then r + ' is bounded near r = if 

and only if A' > P — 2 Z ^ 1, equivalently, A' > P -^ L 2, which means that Cv is in the 

good range with respect to the 6>-stable parabolic subalgebra defined by Cv in the sense 
of Vogan. 

9.8. We end with some remarks and conjectures, primarily concerning the precise form of 
the discrete spectrum and also the continuous spectrum (where it would be very interesting 
to develop the complete Plancherel formula, given our explicit intertwining operator). 

1) (multiplicity free property) 

Each irreducible component in Theorem 9.1 occurs as multiplicity free. If p = 2, then 
w p ' q is the direct sum of an irreducible highest weight module and a lowest one. It 
was proved in [19] that the multiplicity in the full Plancherel formula (both discrete and 
continuous spectrum) is at most one, in the branching law of any highest weight module 
of scalar type with respect to any symmetric pair. 

2) (at most finitely many discrete spectrum, and full discrete spectrum) 
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If p', q',p" and q" satisfy 

(9.8.1) min(p',g") < 1 and min(g' ,p") < 1, 
then the parameter set in Theorem 9.1 is empty, namely, 

A>(p>, q') n A>(q",p") = A'(q\p') fl A\p\ q") = 0. 

We conjecture that there are at most finitely many discrete spectra in the branching law 
zu p,g \G' if (9.8.1) holds. We further conjecture that the full discrete spectrum is as in 
Theorem 9.1 with A'{p, q) replaced by A {p, q) everywhere. 

3) (no discrete spectrum) 

Furthermore, if we exclude the case such as G = 0(p, q—l)x O(l) (see §7.2), namely, 

if 

(9.8.2) min(p', q") < 1, min^p") < 1, p + q > 1, and p" + q" > 1, 
then 

A (p>,q>)nA (q",p") = A (q>,p>)nA (p",q") = 0. 

It is likely that there is no discrete spectra in the branching law ro p ' 9 | G / if (9.8.2) is 
satisfied. 

We note that the condition (9.8.2) is equivalent to that at least one of X(p',q') or 
X(q",p") is a non-compact Riemannian symmetric space and at least one of X(q',p') or 
X(p",q") is a non-compact Riemannian symmetric space. 

4) (discretely decomposable case) 
The opposite extremal case is when 

(9.8.3) min(p',p",q',q") = 0. 

As we have proved in Theorem 4.2, the restriction w p,<1 \g' is discretely decomposable with- 
out any continuous spectrum. We have obtained the full branching formula in Theorem 7.1 
by using Theorem 4.2 and the K-type formula of w p,q . 

If we employ only the method in this section to the special case (9.8.3), then we do 
not have to consider Cases (2) and (3) in §9.7. Then, Theorem 9.1 exhausts all discrete 
spectra in Theorem 7.1 in most cases, but there are some few exceptions. To be precise, 
we consider the case p" = without loss of generality. Then, in view of Theorem 7.1, 
the right side of Theorem 9.1 exhausts all discrete spectra if q" > 5; while at most two of 
(g', K')-modules ^\ Kl 7r°! q x are missing in Theorem 9.1 if q' < 4. The precise missing 
parameters in the case p" = and q' = are: A = ±| (q" = 1); A = 0, 1 (q" = 2); A = \ 
(q" = 3); and A = 1 (q" = 4). In order to to cover all missing parameter by the purely 
geometric method of this section, we should notice a specific feature in the case p" = 0: 

i) M + is a dense subspace of M. 

ii) Any real analytic functions on M + satisfying the Yamabe equation corresponding to 
K'-finite vectors of the ($j', i^')-module with the above missing parameter extend to 
real analytic functions on M (see Corollary 4.3 (2)). 

5) (explicit continuous spectrum) 
We conjecture that 

L 2 -IndpLx(e (g) C V ^ TA ) M L 2 -IndpL x (e <g> C y ^ TA ) (A G R) 

is a continuous spectrum with multiplicity free if min(p',p", q\ q") > 0. 
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